The solution is OK. Just a reminder that these functions are on the critical path and the import statement inside them can slow down the performance.

Thanks for the reminder. Below is a simple test for the performance.

def test_import():
    import warnings
    warnings.filterwarnings("ignore")

    y = 3 + 5
    x = y + 2
    
    return x, y


def test_no_import():
    y = 3 + 5
    x = y + 2
    
    return x, y


if __name__ == "__main__":
    import timeit
    num_calls = int(1e6)
    print(timeit.timeit("test_import()", setup="from __main__ import test_import", number=num_calls))
    print(timeit.timeit("test_no_import()", setup="from __main__ import test_no_import", number=num_calls))

Results on my local machine:

0.6618426890054252
0.06726869300473481

Looks like the defect is acceptable.

The solution is OK. Just a reminder that these functions are on the critical path and the import statement inside them can slow down the performance.

Thanks for the reminder. Below is a simple test for the performance.

def test_import():
    import warnings
    warnings.filterwarnings("ignore")

    y = 3 + 5
    x = y + 2
    
    return x, y


def test_no_import():
    y = 3 + 5
    x = y + 2
    
    return x, y


if __name__ == "__main__":
    import timeit
    num_calls = int(1e6)
    print(timeit.timeit("test_import()", setup="from __main__ import test_import", number=num_calls))
    print(timeit.timeit("test_no_import()", setup="from __main__ import test_no_import", number=num_calls))

Results on my local machine:

0.6618426890054252
0.06726869300473481

Looks like the defect is acceptable.

For non-spark case 0.5s overhead each call is considered as big because it happens many times in the inner loop, not just once.

The solution is OK. Just a reminder that these functions are on the critical path and the import statement inside them can slow down the performance.

Thanks for the reminder. Below is a simple test for the performance.

def test_import():
    import warnings
    warnings.filterwarnings("ignore")

    y = 3 + 5
    x = y + 2
    
    return x, y


def test_no_import():
    y = 3 + 5
    x = y + 2
    
    return x, y


if __name__ == "__main__":
    import timeit
    num_calls = int(1e6)
    print(timeit.timeit("test_import()", setup="from __main__ import test_import", number=num_calls))
    print(timeit.timeit("test_no_import()", setup="from __main__ import test_no_import", number=num_calls))

Results on my local machine:

0.6618426890054252
0.06726869300473481

Looks like the defect is acceptable.

For non-spark case 0.5s overhead each call is considered as big because it happens many times in the inner loop, not just once.

The overhead is for 1e6 calls.